Systems and Methods for Augmented Reality Aware Contents

ABSTRACT

Methods and systems for rendering augmented reality aware standard digital content are disclosed. The method includes detecting a context sensitive trigger initiated in response to activation of a trigger condition related to a standard digital content; determining augmented reality content information associated with the context sensitive trigger using augmented reality awareness data; retrieving based on the augmented reality content information, at least one of augmented reality trigger information, augmented reality digital content or mixed content experience configuration; activating at least one reality source based on the context sensitive trigger to capture reality data in response to retrieving; identifying at least one augmented reality trigger in the at least one reality source; and selectively rendering one of the at least one portion of the standard digital content, the augmented reality digital content and the reality data relative to the at least one portion of the standard digital content.

TECHNICAL FIELD

This disclosure relates generally to augmented reality and moreparticularly to systems and methods for augmented reality awarecontents.

BACKGROUND

Augmented reality is a context aware content presentation technologythat is used to provide situation related assistance. Standard digitalcontent is any type of content that exists in the form of digital dataand is rendered by standard rendering tools that support the particularcontent type. Mixed content experience is the merging of augmentedreality and standard digital contents to produce new dynamic experienceand visualizations where they co-exist and interact in real time.

In one of the conventional augmented reality tool, augmented realitytechnology is used to overlay wireframes, icons, and texts onto the realworld view. Similarly, augmented reality tools try to augment realitywith existing standard digital content. However, in this augmentedreality tool, the standard digital content is completely converted intoaugmented reality based digital content. This process is time, effort,and cost intensive as additional augmented reality content is required.

The above cited problem is addressed in another conventional augmentedreality tool that provides a mechanism for inserting immersive contentsinto eBook for providing augmented reality experience for e-book user.However, this mechanism works only for e-books and has a limitation ofusing widget based augmented reality only and fails to work with otherstandard digital content. Moreover, this augmented reality tool usesonly camera (and media frames) as reality source and input.

SUMMARY

In one embodiment, a method of rendering augmented reality awarestandard digital content is disclosed. The method includes detecting, bya user device, a context sensitive trigger initiated in response toactivation of a trigger condition related to a standard digital content;determining augmented reality content information associated with thecontext sensitive trigger using augmented reality awareness data;retrieving based on rendering capability of the user device andaugmented reality content information, at least one of augmented realitytrigger information, augmented reality digital content associated withthe augmented reality trigger information or mixed content experienceconfiguration; activating at least one reality source based on thecontext sensitive trigger to capture reality data in response toretrieving the augmented reality content information, the augmentedreality trigger information, and the mixed content experienceconfiguration; identifying, by the user device, at least one augmentedreality trigger in the at least one reality source, when the renderingcapability of the user device comprises ability to identify augmentedreality triggers in reality sources; and selectively rendering, by theuser device, one of the at least one portion of the standard digitalcontent, the augmented reality digital content or the reality datacaptured by the at least one reality source relative to the at least oneportion of the standard digital content, based on the renderingcapability of the user device and the mixed content experienceconfiguration.

In another embodiment, a system for rendering augmented reality awarestandard digital content is disclosed. The system includes at least oneprocessors and a computer-readable medium. The computer-readable mediumstores instructions that, when executed by the at least one processor,cause the at least one processor to perform operations that includedetecting a context sensitive trigger initiated in response toactivation of a trigger condition related to a standard digital content;determining augmented reality content information associated with thecontext sensitive trigger using augmented reality awareness data;retrieving based on rendering capability of the user device andaugmented reality content information, at least one of augmented realitytrigger information, augmented reality digital content associated withthe augmented reality trigger information or mixed content experienceconfiguration; activating at least one reality source based on thecontext sensitive trigger to capture reality data in response toretrieving the augmented reality content information, the augmentedreality trigger information, and the mixed content experienceconfiguration; identifying at least one augmented reality trigger in theat least one reality source, when the rendering capability of the userdevice comprises ability to identify augmented reality triggers inreality sources; and selectively rendering one of the at least oneportion of the standard digital content, the augmented reality digitalcontent or the reality data captured by the at least one reality sourcerelative to the at least one portion of the standard digital content,based on the rendering capability of the user device and the mixedcontent experience configuration.

In yet another embodiment, a non-transitory computer-readable storagemedium for rendering augmented reality aware standard digital content isdisclosed, which when executed by a computing device, cause thecomputing device to: detect, by a user device, a context sensitivetrigger initiated in response to activation of a trigger conditionrelated to a standard digital content; determine augmented realitycontent information associated with the context sensitive trigger usingaugmented reality awareness data; retrieve based on rendering capabilityof the user device and augmented reality content information, at leastone of augmented reality trigger information, augmented reality digitalcontent associated with the augmented reality trigger information mixedcontent experience configuration; activate at least one reality sourcebased on the context sensitive trigger to capture reality data inresponse to retrieving the augmented reality content information, theaugmented reality trigger information, and the mixed content experienceconfiguration; identify, by the user device, at least one augmentedreality trigger in the at least one reality source, when the renderingcapability of the user device comprises ability to identify augmentedreality triggers in reality sources; and selectively render, by the userdevice, one of the at least one portion of the standard digital content,the augmented reality digital content or the reality data captured bythe at least one reality source relative to the at least one portion ofthe standard digital content, based on the rendering capability of theuser device and the mixed content experience configuration.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate exemplary embodiments and, togetherwith the description, serve to explain the disclosed principles.

FIG. 1 illustrates a block diagram of an exemplary computer system forimplementing various embodiments.

FIG. 2 illustrates a flowchart of a method of rendering augmentedreality aware standard digital contents, in accordance with anembodiment.

FIG. 3 illustrates a flowchart of a method of rendering augmentedreality aware standard digital contents, in accordance with anotherembodiment.

FIG. 4 illustrates a flowchart of a method of creating augmented realityaware standard digital contents, in accordance with an embodiment.

FIG. 5 is a block diagram illustrating a system of rendering augmentedreality aware contents, in accordance with an embodiment.

DETAILED DESCRIPTION

Exemplary embodiments are described with reference to the accompanyingdrawings. Wherever convenient, the same reference numbers are usedthroughout the drawings to refer to the same or like parts. Whileexamples and features of disclosed principles are described herein,modifications, adaptations, and other implementations are possiblewithout departing from the spirit and scope of the disclosedembodiments. It is intended that the following detailed description beconsidered as exemplary only, with the true scope and spirit beingindicated by the following claims.

Additional illustrative embodiments are listed below. In one embodiment,a block diagram of an exemplary computer system for implementing variousembodiments is disclosed in FIG. 1. Computer system 102 may comprise acentral processing unit (“CPU” or “processor”) 104. Processor 104 maycomprise at least one data processor for executing program componentsfor executing user- or system-generated requests. A user may include aperson, a person using a device such as such as those included in thisdisclosure, or such a device itself. The processor may includespecialized processing units such as integrated system (bus)controllers, memory management control units, floating point units,graphics processing units, digital signal processing units, etc. Theprocessor may include a microprocessor, such as AMD Athlon, Duron orOpteron, ARM's application, embedded or secure processors, IBM PowerPC,Intel's Core, Itanium, Xeon, Celeron or other line of processors, etc.Processor 104 may be implemented using mainframe, distributed processor,multi-core, parallel, grid, or other architectures. Some embodiments mayutilize embedded technologies like application-specific integratedcircuits (ASICs), digital signal processors (DSPs), Field ProgrammableGate Arrays (FPGAs), etc.

Processor 104 may be disposed in communication with one or moreinput/output (I/O) devices via an I/O interface 106. I/O interface 106may employ communication protocols/methods such as, without limitation,audio, analog, digital, monoaural, RCA, stereo, IEEE-1394, serial bus,universal serial bus (USB), infrared, PS/2, BNC, coaxial, component,composite, digital visual interface (DVI), high-definition multimediainterface (HDMI), RF antennas, S-Video, VGA, IEEE 802.n /b/g/n/x,Bluetooth, cellular (e.g., code-division multiple access (CDMA),high-speed packet access (HSPA+), global system for mobilecommunications (GSM), long-term evolution (LTE), WiMax, or the like),etc.

Using I/O interface 106, computer system 102 may communicate with one ormore I/O devices. For example, an input device 108 may be an antenna,keyboard, mouse, joystick, (infrared) remote control, camera, cardreader, fax machine, dangle, biometric reader, microphone, touch screen,touchpad, trackball, sensor (e.g., accelerometer, light sensor, GPS,gyroscope, proximity sensor, or the like), stylus, scanner, storagedevice, transceiver, video device/source, visors, etc. An output device110 may be a printer, fax machine, video display (e.g., cathode ray tube(CRT), liquid crystal display (LCD), light-emitting diode (LED), plasma,or the like), audio speaker, etc. In some embodiments, a transceiver 112may be disposed in connection with processor 104. Transceiver 112 mayfacilitate various types of wireless transmission or reception. Forexample, transceiver 112 may include an antenna operatively connected toa transceiver chip (e.g., Texas Instruments WiLink WL1283, BroadcomBCM4750IUB8, Infineon Technologies X-Gold 618-PMB9800, or the like),providing IEEE 802.11a/b/g/n, Bluetooth, FM, global positioning system(GPS), 2G/3G HSDPA/HSUPA communications, etc.

In some embodiments, processor 104 may be disposed in communication witha communication network 114 via a network interface 116. Networkinterface 116 may communicate with communication network 114. Networkinterface 116 may employ connection protocols including, withoutlimitation, direct connect, Ethernet (e.g., twisted pair 10/100/1000Base T), transmission control protocol/internet protocol (TCP/IP), tokenring, IEEE 802.11a/b/g/n/x, etc. Communication network 114 may include,without limitation, a direct interconnection, local area network (LAN),wide area network (WAN), wireless network (e.g., using WirelessApplication Protocol), the Internet, etc. Using network interface 116and communication network 114, computer system 102 may communicate withdevices 118, 120, and 122. These devices may include, withoutlimitation, personal computer(s), server(s), fax machines, printers,scanners, various mobile devices such as cellular telephones,smartphones (e.g., Apple iPhone, Blackberry, Android-based phones,etc.), tablet computers, eBook readers (Amazon Kindle, Nook, etc.),laptop computers, notebooks, gaming consoles (Microsoft Xbox, NintendoDS, Sony PlayStation, etc.), or the like. In some embodiments, computersystem 102 may itself embody one or more of these devices.

In some embodiments, processor 104 may be disposed in communication withone or more memory devices (e.g., RAM 126, ROM 128, etc.) via a storageinterface 124. Storage interface 124 may connect to memory devices 130including, without limitation, memory drives, removable disc drives,etc., employing connection protocols such as serial advanced technologyattachment (SATA), integrated drive electronics (IDE), IEEE-1394,universal serial bus (USB), fiber channel, small computer systemsinterface (SCSI), etc. The memory drives may further include a drum,magnetic disc drive, magneto-optical drive, optical drive, redundantarray of independent discs (RAID), solid-state memory devices,solid-state drives, etc.

Memory devices 130 may store a collection of program or databasecomponents, including, without limitation, an operating system 132, auser interface application 134, a web browser 136, a mail server 138, amail client 140, a user/application data 142 (e.g., any data variablesor data records discussed in this disclosure), etc. Operating system 132may facilitate resource management and operation of the computer system102. Examples of operating system 132 include, without limitation, AppleMacintosh OS X, Unix, Unix-like system distributions (e.g., BerkeleySoftware Distribution (BSD), FreeBSD, NetBSD, OpenBSD, etc.), Linuxdistributions (e.g., Red Hat, Ubuntu, Kubuntu, etc.), IBM OS/2,Microsoft Windows (XP, Vista/7/8, etc.), Apple iOS, Google Android,Blackberry OS, or the like. User interface 134 may facilitate display,execution, interaction, manipulation, or operation of program componentsthrough textual or graphical facilities. For example, user interfacesmay provide computer interaction interface elements on a display systemoperatively connected to computer system 102, such as cursors, icons,check boxes, menus, scrollers, windows, widgets, etc. Graphical userinterfaces (GUIs) may be employed, including, without limitation, AppleMacintosh operating systems' Aqua, IBM OS/2, Microsoft Windows (e.g.,Aero, Metro, etc.), Unix X-Windows, web interface libraries (e.g.,ActiveX, Java, Javascript, AJAX, HTML, Adobe Flash, etc.), or the like.

In some embodiments, computer system 102 may implement web browser 136stored program component. Web browser 136 may be a hypertext viewingapplication, such as Microsoft Internet Explorer, Google Chrome, MozillaFirefox, Apple Safari, etc. Secure web browsing may be provided usingHTTPS (secure hypertext transport protocol), secure sockets layer (SSL),Transport Layer Security (TLS), etc. Web browsers may utilize facilitiessuch as AJAX, DHTML, Adobe Flash, JavaScript, Java, applicationprogramming interfaces (APIs), etc. In some embodiments, computer system102 may implement mail server 138 stored program component. Mail server138 may be an Internet mail server such as Microsoft Exchange, or thelike. The mail server may utilize facilities such as ASP, ActiveX, ANSIC++/C#, Microsoft .NET, CGI scripts, Java, JavaScript, PERL, PHP,Python, WebObjects, etc. The mail server may utilize communicationprotocols such as internet message access protocol (IMAP), messagingapplication programming interface (MAPI), Microsoft Exchange, postoffice protocol (POP), simple mail transfer protocol (SMTP), or thelike. In some embodiments, computer system 102 may implement mail client140 stored program component. Mail client 140 may be a mail viewingapplication, such as Apple Mail, Microsoft Entourage, Microsoft Outlook,Mozilla Thunderbird, etc.

In some embodiments, computer system 102 may store user/application data142, such as the data, variables, records, etc. as described in thisdisclosure. Such databases may be implemented as fault-tolerant,relational, scalable, secure databases such as Oracle or Sybase.Alternatively, such databases may be implemented using standardized datastructures, such as an array, hash, linked list, struct, structured textfile (e.g., XML), table, or as object-oriented databases (e.g., usingObjectStore, Poet, Zope, etc.). Such databases may be consolidated ordistributed, sometimes among the various computer systems discussedabove in this disclosure. It is to be understood that the structure andoperation of the any computer or database component may be combined,consolidated, or distributed in any working combination.

It will be appreciated that, for clarity purposes, the above descriptionhas described embodiments of the invention with reference to differentfunctional units and processors. However, it will be apparent that anysuitable distribution of functionality between different functionalunits, processors or domains may be used without detracting from theinvention. For example, functionality illustrated to be performed byseparate processors or controllers may be performed by the sameprocessor or controller. Hence, references to specific functional unitsare only to be seen as references to suitable means for providing thedescribed functionality, rather than indicative of a strict logical orphysical structure or organization.

FIG. 2 illustrates a flowchart of a method for rendering augmentedreality aware standard digital contents, in accordance with anembodiment. To this end, first the augmented reality aware standarddigital content is created and includes the standard digital content andaugmented reality awareness metadata. Examples of standard digitalcontent may include, but are not limited to e-books, PDF files, Notepad,images, Web pages, Microsoft Word document, audio content, 3D objects,virtual reality, animated content, and video content.

The augmented reality aware standard digital content is created, suchthat, if a user device does not have capability to render the augmentedreality over the standard digital content, the user device will onlydisplay the standard digital content. In other words, the augmentedreality aware standard digital content is not limited only tospecialized user devices. As a result, creators and users of augmentedreality aware standard digital content do not face an overhead ofkeeping multiple versions of standard digital content, with and withoutthe augmented reality awareness built in. The same content can be usedacross user devices with different rendering capabilities.

The augmented reality aware standard digital content includes thestandard digital content and augmented reality awareness metadata, whichfurther includes augmented reality enablement information and detailsassociated with the items include therein. The augmented realityawareness metadata may be made up of, but is not limited to files,databases or digital stores. It may be stored outside the standarddigital content. Alternatively, at least a part of the augmented realityawareness metadata may be stored inside the standard digital content. Inother words, it may be partially or full stored in the standard digitalcontent.

The augmented reality enablement information is also a metadata thatfurther includes details regarding association between context sensitivetriggers and augmented reality content information. In other words,whenever a particular context sensitive trigger is detected, theaugmented reality content information associated with that particularcontext sensitive trigger is retrieved. Thus, a context sensitivetrigger that does not have an associated augmented reality contentinformation, when detected, will not lead to any further action orresult. In other words, such context sensitive triggers may be ignored.The augmented reality enablement information also includes detailsregarding turning on/off augmented reality awareness in a user device,when a particular context sensitive trigger is detected. In an exemplaryembodiment, the augmented reality enablement information may berepresented using table 1 given below:

TABLE 1 Augmented Reality Context Augmented Enablement Sensitive Realityinformation Trigger Content (ARC) ID (CST) ID Information ID ARawareness ARE1 CST1 ARC1 ON ARE2 CST2 ARC1 OFF ARE3 CST3 ARC2 ON ARE4CST4 ARC2 OFF

Thus, referring to table 1 given above, if context sensitive trigger:CST2 is detected, it is determined that ARC1 is the augmented realitycontent information that is associated with CST2. Additionally, it isalso determined that whenever CST2 is detected, the augmented realityawareness needs to be turned off. Similarly, whenever CST4 is detected,augmented reality awareness is turned off. However, when CST1 and CST3are detected, augmented reality awareness is turned on. This is furtherexplained in detail below.

A context sensitive trigger may be detected by a user device based onactivation of a trigger condition, which may include one or more of, butis not limited to user actions, user gestures, user profile, userpreferences, ambient environment, device location, device orientation,content-flow actions, and device events comprising power saving mode,locked state, and user device sensor values. The context sensitivetrigger corresponds to the context in which the trigger condition wasactivated. By way of an example, rendering frames in a video from time2:00 to 2:15 seconds may act as a context sensitive trigger. By way ofanother example, second paragraph of fifth page in a document reachingtop of a display in a user device may be a context sensitive trigger. Inan exemplary embodiment, the context sensitive triggers may berepresented using the table 2 given below:

TABLE 2 Context Standard Sensitive Digital Trigger Trigger ID Content IDContext Condition CST1 CID1 Focus Page 2 CST2 CID1 Out of Page 2 focusCST3 CID1 Focus Page 3, paragraph 5 CST4 CID1 Out of focus Page 3,paragraph 5 CST5 CID2 Play started Full video CST6 CID2 Play stoppedFull video CST7 CID2 Play reached 2:00 to 2:30 seconds

The augmented reality content information includes details regardingassociation of augmented reality triggers with augmented reality digitalcontent and mixed content experience configuration. An augmented realitytrigger may include, but is not limited to a marker, an object,temperature, time lapse, location, sensor values, and sound. Anaugmented reality digital content may include, but is not limited to 3Dmodels, images, audio, video, animated content, virtual reality, andtext. By way of an example, whenever a particular marker (augmentedreality trigger) in the standard digital content or the reality sourcecomes into view, an audio that is mapped to or associated with themarker, is played in the background. By way of another example, when aparticular object is identified in a standard digital content or realitysource, a video associated with that object is played. By way of yetanother example, a beep may be played, whenever a particular temperatureis reached while the user is interacting with a standard digitalcontent. The temperature may be captured by temperature sensors (realitysource) that may either be built inside the user device or may belocated in vicinity of the user and is in communication with the userdevice. In an exemplary embodiment, augmented reality triggers may berepresented using table 3 given below:

TABLE 3 Augmented Reality Type of Trigger ID Trigger Trigger Value ART1Marker 5 ART2 Object Wheel ART3 Temperature >100 degree Celsius ART4Time lapse >60 seconds ART5 Location 1 km around LatN, LongN ART6Sound >20 dB

In this exemplary embodiment, for ART2: detection of a wheel (object) bya reality source (for example, a camera) is an indicator. Similarly, forART3: detection by a temperature sensor (reality source) thatsurrounding temperature or temperature of the user device is greaterthan 100 degree Celsius acts an indicator. In an exemplary embodiment,association of augmented reality triggers with augmented reality digitalcontent stored as the augmented reality content information berepresented and stored as given in table 4 given below:

TABLE 4 Augmented Augmented Reality reality Content Augmented Digitalinformation Reality Content ID Trigger ID (ARDC) ID ARC1 ART1 ARDC1 ARC2ART1 ARDC1 ARC3 ART2 ARDC2 ARC4 ART2 ARDC3

Augmented reality digital content also includes information regardingits relation/association with one or more portions of a standard digitalcontent and may be represented as given in table 5 below, in anexemplary embodiment:

TABLE 5 Augmented Con- Reality ARDC tent Content Experience ID IDportion action Rotate Translate Scale ARDC1 CID1 Page 2 Overlay 0, 0, 00, 45, 0 1, 1, ARDC2 CID1 Full Open full 1 document screen ARDC3 CID2Page 3, Display 60, 0, 0 120, 50, 0 0.5, paragraph 5 popup 0.5, 05 ARDC4CID2 Page 3, Overlay 0, 0, 90 0, 0, 245 1.25, paragraph 5 1.25, 1.25ARDC5 CID3 Full video Play inline 0, 50, 0 0, 355, 0 1.5, 1.5, 1.5 ARDC6CID3 Full video Play full screen ARDC7 CID4 2:00 to 2:30 Play inline 0,50, 0 0, 355, 0 1.5, seconds 1.5, 1.5

Thus, for example, when the augmented reality digital content (with ARDCID: ARDC3) is identified based on an associated augmented realitytrigger, paragraph five of page three is displayed as a popup andorientation of the popup is further modified based on the detail asmentioned in the table above.

The rules governing which of: parts of standard digital content,augmented reality digital content relative to the parts of the standarddigital content, and/or the reality data captured by a reality source,should be rendered on a user device, is stored as the mixed contentexperience configuration. In other words, the mixed content experienceconfiguration defines what combination of standard digital content,augmented reality digital content, and the reality data has to berendered. It defines the chain of events that leads to a final decisionof rendering. This is further explained in detail below in steps 202 to212. The reality source, for example, may include but is not limited toa camera, microphone, sensor values, and digital content. The senorvalues may be captured from multiple sensors that may include, but arenot limited to pressure sensor, proximity sensor, ambient environmentsensors, gyroscope, accelerometer, and compass. Examples of digitalcontent may include, but are not limited to e-books, PDF files, Notepad,images, Web pages, Microsoft Word document, audio content, 3D objects,virtual reality, animated content, and video content.

After creating augmented reality aware standard digital content, when auser, is interacting, via a user device, with such a standard digitalcontent that is augmented reality aware, the user device, at 202 maydetect a context sensitive trigger that was initiated in response toactivation of a trigger condition related to the standard digitalcontent. The user device's detection of the context sensitive triggerwould depend on rendering capability of the user device. Renderingcapability of the user device includes ability to detect contextsensitive and augmented reality triggers, render augmented realitydigital content, and activate one or more reality sources in response toretrieving an augmented reality trigger.

If the user device does not have built-in ability to detect contextsensitive triggers or does not have an application that can do so, theuser device would merely open the standard digital content and would notdetect that the content is augmented reality aware. Referring to table 2provided above, for example, a user may be reading an e-book (CID1) onhis/her tablet that has ability to detect context sensitive triggers.When the user focusses on page 2 of the e-book, a context sensitivetrigger, i.e., CST1 would be detected. Had the tablet been a standardtablet, it would not have detected the context sensitive trigger.

Thereafter, at 204, based on the context sensitive trigger identified at202, the user device determines the augmented reality contentinformation that is associated with the context sensitive trigger usingaugmented reality awareness information, which includes record of suchassociations. At 206, based on the rendering capability of the userdevice and augmented reality content information, one or more ofaugmented reality trigger information, augmented reality digital contentassociated with the augmented reality trigger information, and mixedcontent experience configuration is retrieved.

Thus, when the user device has determined the augmented reality contentinformation based on its mapping with the context sensitive trigger, itis used to identify which augmented reality trigger and augmentedreality digital content are associated with each other. By way of anexample, referring to table 4, ARC2 is determined at 204 due to itsassociation with the context sensitive trigger detected at 202. The ARC2 indicates that ART1 is associated with ARDC1. By way of anotherexample, when ARC4 is determined at 204, association of ART 2 with ARDC3is indicated. Additionally, mixed content experience configuration isalso retrieved, which includes rules that enable the user device to makea decision on what kind and combination of content (standard digitalcontent, augmented reality digital content, and/or reality data) needsto be rendered on the user device, while the user is interacting withthe standard digital content. As a result, the user would be able toexperience mixed content experience on the user device.

After retrieving, one or more reality sources are activated at 208 basedon the context sensitive trigger. The one or more reality sources maycapture reality data. Examples of the reality sources may include, butare not limited to a camera, microphone, sensor values, and digitalcontent. The senor values may be captured from multiple sensors that mayinclude, but are not limited to pressure sensor, proximity sensor,ambient environment sensors, gyroscope, accelerometer, and compass.Examples of digital content may include, but are not limited to e-books,PDF files, Notepad, images, Web pages, Microsoft Word document, audiocontent, 3D objects, virtual reality, animated content, and videocontent.

When in the augmented reality enablement information, the mapping of acontext sensitive trigger and an augmented reality content informationis associated with augmented reality awareness being turned on, one ormore reality sources may be activated. Similarly, when such mapping isassociated with augmented reality awareness being turned off, one ormore reality sources may be deactivated. Referring to the table 1 ofexemplary embodiment given above, mapping of CST1 with ARC1 isassociated with augmented reality being turned on and mapping of CST2with ARC1 is associated with augmented awareness being turned off. Byway of an example, referring to table 1 and 2, when page 2 of CID1 is infocus, CST1 is detected by the user device. Thereafter, the user devicedetermines that ARC1 is mapped to CST1 in table 1, and such mapping isassociated with augmented awareness being turned on. Similarly, CST2's(page 2 of CID1 out of focus) mapping with ARC1 is associated withaugmented awareness being turned off. Thus, in this scenario, wheneverthe page 2 of CID1 is in focus, a camera observing display of the userdevice and the user is turned on and whenever the page 2 is out offocus, the camera is turned off. For the time the camera is switched on,it captures reality data that includes information on user's interactionwith display of the user device, when the page 2 was in focus.

Thereafter, at 210, the user device identifies one or more augmentedreality trigger in the at least one reality source. The user device willonly identify augmented reality trigger, when the rendering capabilityof the user device comprises ability to identify augmented realitytriggers in reality sources. In case, the user device is an ordinarydevice, the user device will not be able to detect any augmented realitytriggers. In continuation of the example given above and referring totable 3, when the camera is switched on and the page 2 is in view of thecamera, it may identify a wheel drawn on the page 3 and thus theaugmented reality trigger ART2 (given in table 3) is identified.

At 212, the user device selectively renders one of one or more portionsof the standard digital content, the augmented reality digital contentand the reality data captured by the one or more reality sourcesrelative to the one or more portions of the standard digital content.The selective rendering is performed based on the mixed contentexperience configuration.

In continuation of the example given above, when ART2 given in table 3is identified, it is further determined that ART2 is associated ormapped to ARDC2 as given in table 4. Thus, when the camera detects awheel on page 2 of CID1, ART2 is identified and that further leads tothe conclusion that ARDC2 needs to be rendered on the user device.Referring now to table 5 given above, rendering ARDC2 translates toopening the complete document for CID1 in full screen on the userdevice. This chain of events, starting with opening of the standarddigital content (that is augmented reality aware) till opening of thecomplete document for CID1 in full screen is one of rules built in themixed content experience configuration.

By way of another example, when a document is being displayed on theuser device and when a specific page comes into view, the augmentedreality awareness is turned on by opening camera. Thereafter, based onwhat is seen by the camera, additional information/overlay inline inthat page/on a separate page may be displayed. By way of yet anotherexample, a part of a huge video (for instance from time 2:00 to 2:15seconds) may be retrieved and displayed on the user device, when aparticular object is found in a camera frame.

The selective rendering on the user device also depends on the renderingcapability of the user device. In other words, if the user device ismerely capable of rendering standard digital content, it will onlyrender the standard digital content on the user device. If the userdevice also has the ability to render augmented reality digital content,the user device would render the augmented reality digital content inrelation to the standard digital content. However, it won't be able toactivate reality sources and thus won't render augmented reality digitalcontent relative to the standard digital content based on the realitydata captured by the reality sources.

By way of an example, when an augmented reality aware PDF document isopened using a normal PDF reader on the user device, the PDF documentgets displayed as regular PDF document without augmented realityawareness. However, when the same document is opened using an augmentedreality aware PDF reader on the user device, additional augmentedreality aware features are rendered as well.

Thus, augmented reality aware standard digital content that provides animmersive mixed content experience is rendered to a user based on ausage context (purpose) and dynamically selected portions of a standarddigital content based on the usage context. Augmented reality andnon-augmented reality content that may superimpose or replace existingnon-augmented reality content is provided, which enables supplementingskills of a maintenance personnel. Moreover, all the reality inputs ofaugmented reality experience are considered to achieve this.

FIG. 3 illustrates a flowchart of a method for rendering augmentedreality aware standard digital contents, in accordance with anotherembodiment. At 302, augmented reality aware standard digital content iscreated. This has been explained in detail in conjunction with FIG. 2.While creating the augmented reality aware standard digital content,both the complete augmented reality awareness and augmented realitydigital content may be configured inline inside the standard digitalcontent. Alternatively, the augmented reality digital content may beconfigured outside the standard digital content but augmented realityawareness may be configured inline inside the standard digital content.In another scenario, both the complete augmented reality awareness andaugmented reality digital content may be configured outside of thestandard digital content.

Thereafter, at 304, a context sensitive trigger that is initiated inresponse to activation of a trigger condition related to a standarddigital content is detected. At 306, augmented reality contentinformation associated with the context sensitive trigger is determinedusing augmented reality awareness data. Based on the augmented realitycontent information and rendering capability of the user device, one ormore of augmented reality trigger information, augmented reality digitalcontent associated with the augmented reality trigger information, andmixed content experience configuration are retrieved at 308. This hasbeen explained in detail in conjunction with FIG. 2.

Thereafter, at 310, one or more reality sources are activated based onthe context sensitive trigger to capture reality data in response toretrieving at 308. At 312, one or more augmented reality triggers areidentified in the one or more reality sources. This has been explainedin detail in conjunction with FIG. 2. At 314, the one or more realitysources are deactivated on expiration of the context sensitive triggeror on identification of a deactivating context sensitive trigger. Inother words, augmented reality awareness may be turned on or off. By wayof an example, when page 3 of a standard digital content comes intoview, the object detection engine may be turned on and when the page 5goes out of view, the object detection engine may be turned off. By wayof another example, while frames from 3:00 to 3:20 seconds are beingrendered, a noise detection engine may be turned on and when the userclicks on pause button of the video, the noise detection engine may beturned off.

The selection of augmented reality engines is also influenced bymultiple factors that may include, but are not limited to userpreferences, augmented reality digital content, augmented realityawareness content and capabilities of the user device. By way of anexample, page 2 in a PDF file may have a marker that is intended to turnon augmented reality in order to find out the name of a building. Inthis case, running only the Optical character recognition (OCR)sub-engine might be faster and thus may be executed to save time andresources.

At 316, the user device, automatically determines the extent ofrendering the augmented reality digital content on the user device basedon capabilities of the user device. In an embodiment, capabilities ofthe user device are queried to decide on suitable extent of rendering.The capabilities may include, but are not limited to processing power,available memory, display size, remaining battery, and communicationspeed. These capabilities of the user device are used in variousdecision factors considered by augmented reality aware contentsrenderer. These decision factor may include, but are not limited toselection of suitable augmented reality engines, selection ofaugmentation objects, and selection of quality of augmentation.

By way of an example, a user device having 2 Ghz processor speed may runX augmented reality engine, whereas a user device having 1 GHz processorcan run a Y augmented reality engine. The capabilities of X augmentedreality engine are superior to that of Y augmented reality engine. Byway of another example, a user device may have a hardware accelerated 3Dengine, whereas another user device may not have the hardwareaccelerated 3D engine. Thus, the decision would be made not to display3D augmentations on the user device that does not have the hardwareaccelerated 3D engine.

Thereafter, at 316, the use device selectively renders one of the one ormore portions of the standard digital content, the augmented realitydigital content, and the reality data captured by the one or morereality sources. This has been explained in detail in conjunction withFIG. 2.

FIG. 4 illustrates a flowchart of a method for creating augmentedreality aware standard digital content, in accordance with anembodiment. At 402, associations between context sensitive triggers andaugmented reality content information and association between augmentedreality trigger information and augmented reality digital content isformed. Thereafter, at 404, the mixed content experience configurationthat includes rules governing the selective rendering are configured.This has been explained in detail in conjunction with FIG. 2.

FIG. 5 is a block diagram illustrating an augmented reality renderingsystem 500 for rendering augmented reality aware standard digitalcontents, in accordance with an embodiment. Augmented reality renderingsystem 500 includes a configurator module 502, a content store module504, and a rendering module 506.

Configurator module 502 is used by a content creator 508 to create andconfigure augmented reality aware standard digital content usingexternal standard digital content that is fed into configurator module502. Configurator module 502 stores and retrieves standard digitalcontent and Augmented Reality Aware Metadata (ARAMA) information from/toa content access controller 510 in content store module 504 that stores,manages provides augmented reality aware standard digital content torendering module 506. To enable this, control access controller 510communicates with a content processor 512 in rendering module 506.Content processor 512 then makes this data available to various modulesand components in rendering module 506. The creation and configurationof augmented reality aware standard digital content has been explainedin detail in conjunction with FIG. 2.

The rendering module 506 further includes a context sensitive triggermodule 514, a reality source controller 516, a content mixer 518, a useraction processor 520, and a content rendering module 522. Contextsensitive trigger module 514 receives context inputs from other modulesand components within rendering module 506. It analyzes and generatescontext sensitive triggers and makes them available to other modules andcomponents. Context sensitive triggers have been explained inconjunction with FIG. 2.

Reality source controller 516 receives reality data captured by thereality sources. Examples of reality sources may include but are notlimited to a camera, sensor values, and digital content. The realitydata may come from the same device and/or other external sources andenvironments. The content captured or generated by content processor512, context sensitive trigger module 514, and reality source controller516 is communicated to content mixer 518.

Content mixer 518 that includes an augmented reality engine 524 and amixed content engine 526, uses these engines to mix the content receivedand sends the mixed content to content rendering module 522. In additionto receiving the mixed content, content rendering module 522 receivesinputs from user action processor 520 that receives inputs from acontent user 528. Based on the received information, content renderingmodule 522 selectively renders received contents on output devices 530,which are then consumed by content user 528. This has been explained indetail in conjunction with FIG. 2.

Various embodiments of the invention provide systems and methods foraugmented reality aware contents. Augmented reality aware standarddigital content that provides an immersive mixed content experience isrendered to a user based on a usage context (purpose) and dynamicallyselected portions of a standard digital content based on the usagecontext. Augmented reality and non-augmented reality content that maysuperimpose or replace existing non-augmented reality content isprovided, which enables supplementing skills of a maintenance personnel.Moreover, all the reality inputs of augmented reality experience areconsidered to achieve this.

The specification has described systems and methods for augmentedreality aware contents. The illustrated steps are set out to explain theexemplary embodiments shown, and it should be anticipated that ongoingtechnological development will change the manner in which particularfunctions are performed. These examples are presented herein forpurposes of illustration, and not limitation. Further, the boundaries ofthe functional building blocks have been arbitrarily defined herein forthe convenience of the description. Alternative boundaries can bedefined so long as the specified functions and relationships thereof areappropriately performed. Alternatives (including equivalents,extensions, variations, deviations, etc., of those described herein)will be apparent to persons skilled in the relevant art(s) based on theteachings contained herein. Such alternatives fall within the scope andspirit of the disclosed embodiments.

Furthermore, one or more computer-readable storage media may be utilizedin implementing embodiments consistent with the present disclosure. Acomputer-readable storage medium refers to any type of physical memoryon which information or data readable by a processor ay be stored. Thus,a computer-readable storage medium may store instructions for executionby one or more processors, including instructions for causing theprocessor(s to perform steps or stages consistent with the embodimentsdescribed herein. The term “computer-readable medium” should beunderstood to include tangible items and exclude carrier waves andtransient signals, i.e., be non-transitory. Examples include randomaccess memory (RAM), read-only memory (ROM), volatile memory,nonvolatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, andany other known physical storage media.

It is intended that the disclosure and examples be considered asexemplary only, with a true scope and spirit of disclosed embodimentsbeing indicated by the following claims.

What is claimed is:
 1. A method of rendering augmented reality awarestandard digital content, the method comprising: detecting, by a userdevice, a context sensitive trigger initiated in response to activationof a trigger condition related to a standard digital content;determining augmented reality content information associated with thecontext sensitive trigger using augmented reality awareness data;retrieving based on rendering capability of the user device andaugmented reality content information, at least one of augmented realitytrigger information, augmented reality digital content associated withthe augmented reality trigger information or mixed content experienceconfiguration; activating at least one reality source based on thecontext sensitive trigger to capture reality data in response toretrieving the augmented reality content information, the augmentedreality trigger information, and the mixed content experienceconfiguration; identifying, by the user device, at least one augmentedreality trigger in the at least one reality source, when the renderingcapability of the user device comprises ability to identify augmentedreality triggers in reality sources; and selectively rendering, by theuser device, one of the at least one portion of the standard digitalcontent, the augmented reality digital content or the reality datacaptured by the at least one reality source relative to the at least oneportion of the standard digital content, based on the renderingcapability of the user device and the mixed content experienceconfiguration.
 2. The method of claim 1, wherein the trigger conditionfor the context sensitive trigger comprises at least one of useractions, user gestures, user profile, user preferences, ambientenvironment, device location, device orientation, content-flow actionsor device events comprising power saving mode, locked state, and userdevice sensor values.
 3. The method of claim 1, wherein the standarddigital content comprises at least one of e-books, PDF files, Notepad,images, Web pages, Microsoft Word document, audio content, 3D objects,virtual reality, animated content or video content.
 4. The method ofclaim 1, wherein the context sensitive trigger corresponds to thecontext in which the trigger condition was activated, the contentsensitive trigger indicates enablement or disablement of an augmentedreality digital content.
 5. The method of claim 1 further comprisingrendering the at least one portion of the standard digital content onthe user device based on the rendering capability of the user device. 6.The method of claim 1 further comprising rendering the at least oneportion of the standard digital content and the augmented realitydigital content on the user device based on the rendering capability ofthe user device, wherein the augmented reality digital content isoverlaid on the standard digital content.
 7. The method of claim 1,wherein the at least one reality source comprises a camera, sensorvalues, and digital content.
 8. The method of claim 1 further comprisingautomatically determining extent of rendering the augmented realitydigital content on the user device based on capabilities of the userdevice.
 9. The method of claim 1, wherein the augmented reality triggercomprises at least one of a marker, an object, temperature, time lapse,location, sensor values or sound.
 10. The method of claim 1 furthercomprising creating the augmented reality aware standard digitalcontent, creating comprising: forming associations between contextsensitive triggers and augmented reality content information,association between augmented reality trigger information and augmentedreality digital content; and configuring the mixed content experienceconfiguration comprising rules governing the selective rendering of atleast one of the at least one portion of the standard digital content,the augmented reality digital content relative to the at least oneportion of the standard digital content or the reality data captured bythe at least one reality source.
 11. The method of claim 1 furthercomprising deactivating the at least one reality source on expiration ofthe context sensitive trigger or identification of a deactivatingcontext sensitive trigger.
 12. The method of claim 1, wherein therendering capability of the user device comprises ability to: detectcontext sensitive triggers, detect augmented reality triggers, renderaugmented reality digital content, and activate the at least one realitysource in response to retrieving an augmented reality trigger.
 13. Asystem for rendering augmented reality aware standard digital content,the system comprising: at least one processors; and a computer-readablemedium storing instructions that, when executed by the at least oneprocessor, cause the at least one processor to perform operationscomprising: detect a context sensitive trigger initiated in response toactivation of a trigger condition related to a standard digital content;determine augmented reality content information associated with thecontext sensitive trigger using augmented reality awareness data;retrieve based on rendering capability of the user device and augmentedreality content information, at least one of augmented reality triggerinformation, augmented reality digital content associated with theaugmented reality trigger information or mixed content experienceconfiguration; activate at least one reality source based on the contextsensitive trigger to capture reality data in response to retrieving theaugmented reality content information, the augmented reality triggerinformation, and the mixed content experience configuration; identify atleast one augmented reality trigger in the at least one reality source,when the rendering capability of the user device comprises ability toidentify augmented reality triggers in reality sources; and selectivelyrender one of the at least one portion of the standard digital content,the augmented reality digital content or the reality data captured bythe at least one reality source relative to the at least one portion ofthe standard digital content, based on the rendering capability of theuser device and the mixed content experience configuration.
 14. Thesystem of claim 13, wherein the trigger condition for the contextsensitive trigger comprises at least one of user actions, user gestures,user profile, user preferences, ambient environment, device location,device orientation, content-flow actions or device events comprisingpower saving mode, locked state, and user device sensor values.
 15. Thesystem of claim 13, wherein the operations further comprise renderingthe at least one portion of the standard digital content on the userdevice based on the rendering capability of the user device.
 16. Thesystem of claim 13, wherein the operations further comprise renderingthe at least one portion of the standard digital content and theaugmented reality digital content on the user device based on therendering capability of the user device, wherein the augmented realitydigital content is overlaid on the standard digital content.
 17. Thesystem of claim 13, wherein the operations further compriseautomatically determining extent of rendering the augmented realitydigital content on the user device based on capabilities of the userdevice.
 18. The system of claim 13, wherein the operations furthercomprise creating the augmented reality aware standard digital content,creating further comprises operations of: forming associations betweencontext sensitive triggers and augmented reality content information,association between augmented reality trigger information and augmentedreality digital content; and configuring the mixed content experienceconfiguration comprising rules governing the selective rendering of atleast one of the at least one portion of the standard digital content,the augmented reality digital content relative to the at least oneportion of the standard digital content or the reality data captured bythe at least one reality source.
 19. The system of claim 13, wherein theoperations further comprise deactivating the at least one reality sourceon expiration of the context sensitive trigger or identification of adeactivating context sensitive trigger.
 20. A non-transitorycomputer-readable storage medium for rendering augmented reality awarestandard digital content, when executed by a computing device, cause thecomputing device to: detect, by a user device, a context sensitivetrigger initiated in response to activation of a trigger conditionrelated to a standard digital content; determine augmented realitycontent information associated with the context sensitive trigger usingaugmented reality awareness data; retrieve based on rendering capabilityof the user device and augmented reality content information, at leastone of augmented reality trigger information, augmented reality digitalcontent associated with the augmented reality trigger information ormixed content experience configuration; activate at least one realitysource based on the context sensitive trigger to capture reality data inresponse to retrieving the augmented reality content information, theaugmented reality trigger information, and the mixed content experienceconfiguration; identify, by the user device, at least one augmentedreality trigger in the at least one reality source, when the renderingcapability of the user device comprises ability to identify augmentedreality triggers in reality sources; and selectively render, by the userdevice, one of the at least one portion of the standard digital content,the augmented reality digital content or the reality data captured bythe at least one reality source relative to the at least one portion ofthe standard digital content, based on the rendering capability of theuser device and the mixed content experience configuration.